feat(snapshot): add cow memory and resident pause lifecycle - #1537
Merged
appcypher merged 34 commits intoSep 10, 2026
Conversation
Use published msb_krun 0.1.33 in place of the superseded stack override. Add atomic, pinned, read-only memory cache realizations keyed by complete manifests. Cover warm reuse, concurrent publication, failed construction, alignment, eviction, and unlink lifetime without exposing CoW policy yet. Prefer .msnap in CLI help and SDK documentation while preserving explicit filenames and content-detected legacy archives. Exercise eight encoding and suffix combinations through direct capture and child staging. This is stack-8 groundwork; public CoW and pause/resume integration and end-to-end qualification remain pending.
Let capture borrow an executor-owned pause generation and workload latch, validate the execution boundary, and leave that source paused on success or failure. Keep the existing running-source call path unchanged. Public pause ownership is not wired yet: agent freezer capability and partial-failure replies must be disambiguated before basic-pause fallback can be exposed safely. Existing checkpoint tests and runtime checks pass; the borrowed-pause path still needs end-to-end qualification.
Record latch ownership before freezer writes and retain it after failed freeze or thaw acknowledgement. Only confirmed thaw releases the latch. Add optional attempt-scoped error details without a protocol generation bump. Treat older ambiguous errors and transport failures as requiring recovery, and fence mutations if acknowledged recovery fails. Cover failure injection, retry ownership, protocol compatibility, and host recovery disposition. Public pause/resume wiring remains pending.
Wire explicit memory policy and pause/resume through the CLI and Rust, Python, Node, and Go SDKs. Preserve user pause ownership across full captures and correct the guest clock before releasing workloads. Restore from pinned immutable memory caches and prepare repeated capture caches from immutable generations, reusing filesystem reflinks where available. Resolve control endpoints and cache paths through the owning local backend, and pin the matching runtime and firmware companions. Add live lifecycle smoke matrices and record Linux/macOS results and remaining qualification gaps. Windows CoW remains unsupported; private zeroing on unplug does not yet reclaim host RAM. This commit does not claim full stack-8 qualification.
Extend the live lifecycle runner with optional memory resizing, unsupported CoW admission checks, archive retention diagnostics, and explicit timeout evidence. Assert retained child contents after archive removal and source shutdown. Record passing Linux capacity convergence and Windows post-restore command timeouts without treating activation latency as usable restore performance. Keep incomplete qualification and implementation work explicit in the report.
|
wow @appcypher this is a good one. This will be useful for the runmsb also |
Capture device workers before interrupt-controller state and RAM so late device completions cannot leave the checkpoint's queues and IRQs at different boundaries. Pin libkrun's processor, interrupt-controller and clock-state fixes. Add portable delayed-restore clock and offline-CPU live fixtures, and wait for application initialization in the lifecycle smoke runner. Record passing Windows ARM64 and nested Linux ARM64 live matrices, plus macOS and Linux x86 regressions. Keep native Windows x86 and Windows CoW cache qualification limits explicit.
Avoid repeated closure validation and disposable staging syncs while retaining payload admission, persistent disk durability and guest activation ordering. Serialize CoW cache misses without serializing warm hits. Preserve sparse archive members with long names, buffer archive extraction, and merge incremental memory extents with a consuming sweep. Resolve flat snapshot image metadata without materializing unused OCI disks and allow those sandboxes to restart without a VMDK. Add correctness regression coverage and record release-build macOS benchmarks, old/new archive reader checks, empty-cache restoration, concurrent CoW restores and child isolation tests.
Create independent local children through the existing capture and restore paths without publishing a durable full snapshot. Reuse immutable RAM generations with reflinks and dirty ranges, and protect handoff and VM backing lifetimes with independent pins. Expose branching across the CLI and Rust, Python, TypeScript and Go SDKs. Replace the unreleased memory-snapshot creation policy with restore-only forked selection, and reject unsupported or incomplete restore requests instead of falling back to a fresh boot. Update documentation and add macOS live lifecycle, isolation, ownership and release benchmark harnesses. macOS release and SDK qualification passed; Linux and Windows qualification remains outstanding.
Enable Windows private RAM backing with protected cache entries and cross-platform lifetime pins. Persist pending restore intent until creation succeeds so failed children cannot cold-boot or mutate sealed snapshot layers through later lifecycle operations. Pin the published native memory and ARM64 interrupt/barrier fixes. Add failed-restore and retained-sibling timer regressions, port ownership checks to Windows, and document the four-host live qualification results.
Expose disk-only capture for running and user-paused managed and flat sandboxes through the existing snapshot CLI and shared SDK paths. Serialize root rollover in runtime control without capturing RAM or advancing its baseline, preserve user pauses, and capability-gate older runtimes. Keep stopped lifecycle locking and direct archive publication. Update API documentation and record live flat/layered coverage on macOS ARM64, Linux x86-64, and Windows ARM64, including active writes, source isolation, direct archives, and mixed disk/full capture sequences.
Replace the development libkrun and vm-memory Git patches with the published msb_krun 0.1.34 family and msb-vm-memory 0.18.0-msb.2. Update imago to 0.1.7 so disk and runtime code share one memory crate. Retain the existing firmware pin and the #7 stack base. Record release validation, 32 checkpoint and 39 snapshot regressions, and the successful Mac live branch/restore/growth/compaction run.
Pipeline capture persistence with two writers and three reusable 32 MiB packs. Reuse runtime-owned immutable RAM object receipts across captures and batch directory durability before the existing root-last publication. Pipeline eager restoration and cold memory backing construction with bounded verified buffers. Seed root journals from unchanged admitted disk files, retaining at most 32 file handles and hashing copied or rewritten layers. Keep SDK payload verification unchanged. Use one authoritative pause/resume mutation and validate its resulting state. Combine cgroup notifications with bounded state rechecks so kernel notification throttling cannot impose an avoidable 10 ms wait. Preserve snapshot formats, dirty-baseline ordering, clock-before-thaw, durability and failure cleanup. Add work counters and regression tests for corruption, cancellation, concurrent writers and low descriptor budgets.
Extend --since exports to omit RAM objects supplied by the explicit base alongside the physical disk prefix. Preserve complete memory maps and CPU/device metadata, and resolve borrowed objects into owned staging for both snapshot load and direct archive restore. Keep --last-layers limited to disk selection. Update CLI/SDK documentation and add malformed-input, corruption, retention and twelve-generation archive regressions, plus a reproducible live qualification harness. Validate 44 snapshot tests on both macOS and Linux and live twelve-step chains for flat, layered and tmpfs roots with eager and forked restores. Record timings and qualification limits in the smoke report. BREAKING CHANGE: replace the unreleased disk-only archive dependency encoding with msb-snapshot-dependencies-v1, without a compatibility shim. Standalone archives and snapshot descriptors remain unchanged.
|
Too many files changed for review (176 files, 100 file limit). Bypass the limit by tagging |
Remove automatic guest-wide sync from execution-state capture while retaining workload freezing, host disk durability and clock-before-thaw ordering. Document the crash-consistent disk-only extraction contract. Use bounded sub-millisecond freezer rechecks, a lightweight CLI executor, and WAL-aware read-only control lookups with existing installation and schema gates. Reuse ambient configuration and route Go control lookups through the same path. Add dirty-memory checkpoint coverage and the macOS/Linux release report, including 480 measured lifecycle cycles and flat/layered restore checks.
Admit operations through short-lived shared-handle locks so pending exec, filesystem I/O and shutdown waits cannot block pause or resume. Preserve consumed-handle rejection while admitted operations retain ownership. Decode filesystem error envelopes before their payloads and propagate paused-runtime diagnostics through unary and streaming operations. Add ownership, cancellation and response-decoding tests, six live concurrency cases on macOS and Linux, and the release timing report. Both hosts passed all live cases and 137 Node unit tests; focused Rust tests, TypeScript checks and native linting also passed.
Add group-scoped names and immutable member directories while preserving portable snapshot identities across independent imports. Advance heads only through known ancestry and retain concurrent sibling captures. Carry source ancestry through capture, restore and local branch. Serialize publication and retain cursor locks during cancellation. Expose group selection and import options across the CLI and Rust, Python, Node and Go SDKs. Update the path-keyed index, archive aliases and downgrade checks. Cover group publication, duplicate imports, archive dependencies and legacy artifact paths with automated tests and isolated macOS live matrices for managed, flat and tmpfs roots. BREAKING CHANGE: Bare snapshot names now select group heads. Address exact members as group:member or use an explicit artifact path. Installed group members cannot be overwritten with force.
Promote Branch & Snapshot with a fork icon and concise CLI examples.\nShow live branching and durable full snapshots in separate code groups.
Use app:saved to select the exact checkpoint under the source sandbox's snapshot group, matching the current restore selector API.
Resolve disk and RAM dependencies across supplied archives and the selected destination group, with an optional external base. Validate complete owned closures before publishing members and select a group head independently of archive order. Expose batch loading through the CLI and Rust, Python, TypeScript, and Go SDKs. Prefer .msb archive names throughout examples and documentation without changing content-based decoding or archive schemas. Record automated coverage and 102 macOS live command checks for full and disk-only batches, including eager and forked restoration. Keep the existing disk-only incremental-export limitation explicit. BREAKING CHANGE: snapshot load takes one or more positional archives; use --dest DIR instead of a positional destination directory.
Use --from-sandbox for snapshot capture, matching the existing SDK source-sandbox naming. Update README, documentation, group examples, and the branch, incremental, disk, and restore smoke scripts. Test required source selection and rejection of --from, including full capture, direct archives, and generated group members. All 13 snapshot parser tests pass. BREAKING CHANGE: msb snapshot create requires --from-sandbox instead of --from. No compatibility alias is provided.
Add a fast host-side snapshot/checkpoint test recipe and a bounded live smoke suite for managed and flat disks. Cover branching, paused capture, disk-only capture, grouped archive loading, eager/forked restore, and source/child isolation. Verify runtime exit and reclaim successful test homes while retaining reports and failure diagnostics. Wire the smoke suite into Linux/KVM CI, document the commands, align existing SDK/CLI tests with snapshot groups, and remove the extra branching slogan from the README.
Merge the updated #7 parent into #8, preserving release transport, runtime bootstrap, SDK lifecycle, and migration ordering. Record the reviewed conflict resolutions without importing unrelated main changes. Add bounded private freeze/thaw routing over the existing transport, complete-frame capture barriers, retained input accounting, and restore metadata validation. Keep generation 8 unchanged and replace only the unreleased full-checkpoint contract. Preserve exact run ownership, lineage publication, memory-cache pins, and failed-start cleanup. Include focused regression tests, isolated live-test harnesses, and the Mac/Linux qualification report. Native runtime and guest tests pass, but retained-stdin control-credit starvation and mixed-load throughput qualification remain open. This is a progress checkpoint, not a claim that #8 is merge-ready.
Pin msb_krun and msb_krun_utils to the published 0.1.35 release and refresh only their matching lockfile entries. Pick up runtime resolution of optional macOS GIC APIs without changing the snapshot format, guest protocol, or stack ancestry.
Keep stdio guard construction in the I/O error domain so the runtime spawn helper compiles on Windows. Own successful flag changes immediately and roll them back if a later handle update fails. Add Windows regression tests for partial failure, duplicate handles and unchanged non-inheritable handles, and run them in both Windows CI jobs. Account for Unix-only control-link parameters on Windows so the strict Clippy gate remains usable.
Separate retained input debt from metadata credit and allow bounded independent control progress without violating stream or lifecycle fences. Yield guest reads by actual work, preserve TCP half-close tails, and retain shared-memory payload ownership on combined transports. Refresh maintenance clocks at transport admission, keep unrelated cleanup from fencing fresh commands, and strengthen source/child readiness checks under autonomous input pressure. Refuse incompatible unreleased transport barriers explicitly. Validated the frozen patch with macOS and Linux live transport suites and benchmarks, plus protocol, guest, runtime and harness regression tests. Final stack integration with the newer registry dependency is checked separately.
Merge the updated immediate parent while retaining both Windows lifecycle-handoff and stdio-inheritance regression tests. Resolve the duplicate Windows endpoint-argument guard without changing its existing behavior. Carry the release baseline, exact stack CI triggers and CI corrections forward. Formatting and diff checks pass; no unrelated main changes are included.
Merge the preserved transport fix commit onto the reconciled #8 branch. Its fourteen files are byte-for-byte identical to the macOS/Linux-qualified patch; retain the newer msb_krun 0.1.35 registry pin and both Windows test sets from the current stack.
Run all Windows-specific spawn regressions in the existing quality lane so the lifecycle-handoff cases preserved during stack reconciliation are executed alongside stdio inheritance cleanup. No runtime behavior changes.
Use the shared snapshot resolver from Python, retaining async not-found errors and covering group, member, identity, directory and archive inputs. Avoid the deprecated musl time_t alias without rounding freezer waits. Use mirrored test images so registry authentication cannot hide regressions.
Integrate the reviewed parent fixes without changing branch, paused-source, or root-disk growth semantics. Carry source recovery diagnostics through final group and archive publication, and expose only confirmed artifact paths through the SDK errors. Validate final group paths, head selection, ancestry cursors, artifact indexing, backend routing, and publication failures. The SDK suite passes 709 tests with two ignored; production Clippy and formatting pass.
Inherit the two parent binding lint fixes while retaining group-aware async snapshot resolution and unchanged Python keyword arguments. Final Python Clippy passes with warnings denied.
Keep the short /tmp default while allowing a separate short home parent on hosts with quota-limited tmpfs. Each test still creates its own isolated home and never borrows existing sandbox state. Add a fixture regression for directory ownership and preserve the concurrent README update. All 34 transport harness unit tests pass.
|
@appcypher good job getting this job done, with this landed, I could go ahead and implement the pause and resume in the oci runtime right ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Add direct local branching, explicit CoW full-snapshot restore, resident pause/resume, and preferred
.msbarchive naming as stack item #8. The September 9 platform report records fixes for the earlier Windows ARM64 restore hang and private-memory gaps. Final integrated-head qualification is tracked separately below; Windows x86-64 remains untested.Description
appcypher/live-root-disk-growth) without importing unrelated Microsandbox main changes.msb branch SOURCE --name CHILDand matching SDK methods for independent local children without publishing a durable full snapshot. A running source resumes after capture; a user-paused source stays paused.--forkedfor private CoW memory on supported Linux, macOS, and Windows hosts, backed by a protected immutable cache with retained file ownership and reflink reuse where available. No source creation-time memory option is required; the superseded unreleased memory-snapshot API is removed..msbin help and documentation while retaining content-based archive detection, legacy suffixes, explicit filenames, and plain-tar output.msb_krun/msb_krun_utils0.1.35,msb-vm-memory0.18.0-msb.2, andmsb-imago0.1.7. Remove the development Cargo Git patches; retain the matching libkrunfw submodule pin.scripts/smoke/reports/cow-platform-fixes-2026-09-09.md: eager/forked restore and direct branching pass on macOS ARM64, Linux x86-64, Linux ARM64/nested KVM, and Windows ARM64. These results apply to their recorded revisions, not a fresh full-platform qualification of 0.1.35 or subsequent integration changes.msb snapshot create NAME --from-sandbox SOURCE, including direct archive capture, without capturing RAM. It is crash-consistent; seescripts/smoke/reports/live-disk-snapshot-2026-09-09.mdfor the three-host flat/layered checks.--groupselects another, andgroup:memberselects an exact checkpoint. Bare groups resolve their selected heads; multi-archive imports resolve dependencies within the batch and explicitly selected group.--since(disk layers and reused RAM objects), disk-layer selection with--last-layers, explicit-base restore, and manual live/stopped disk compaction. Successive disk-only capture/export still has the documented layer-ID limitation; full-checkpoint incremental exports are unaffected.CLI examples
Branch a running sandbox
branchcaptures RAM, disk, CPU and device state with a brief source pause, then starts a detached child. CoW is inherent: no--forkedoption is needed. It writes local backing but does not create a registered, durable full snapshot. Branching requires a new child name and currently rejects sources with published host ports.Branch a paused source
pauseandresumekeep the VM resident; neither creates a snapshot.Save a durable full snapshot, then restore children
The snapshot is installed in group
source, so its exact selector issource:saved.--forkedselects the restored child's memory backing, not the snapshot's contents. It requires a full snapshot and cannot be combined with a fresh image boot or--disk-only.Capture and restore directly through an archive
Export changes and compact explicitly
After compaction, export a new standalone baseline before exporting further deltas. This does not invalidate previously captured snapshots.
SDK examples
Each language also exposes branching on sandbox handles returned by
get. The following examples assume an existing sandbox namedsource; full-restore examples usesource:saved, the grouped snapshot created above.Rust, inside an async fallible context:
Python, inside an async function:
TypeScript:
Go, using
mforgithub.com/superradcompany/microsandbox/sdk/goand an existing context:Stop the created children explicitly when finished; branching creates detached children.
Test Plan — historical evidence
The checked items below record prior runs at their documented revisions. They are not a claim that the final integrated branch with libkrun 0.1.35 and the latest transport changes has completed every test.
scripts/smoke/reports/registry-dependencies-2026-09-09.md; Linux/Windows live matrices were not rerun for this packaging update.scripts/smoke/cli/direct-branch.pyandscripts/smoke/cli/branch-ownership.py.--forkedCLI matrices on Linux x86-64, Linux ARM64/nested KVM, and Windows ARM64; backing lifetime, failed-restore refusal, sibling isolation, and timer-progress regressions. Seecow-platform-fixes-2026-09-09.mdfor exact coverage and limitations.cargo fmt --all -- --checkandgit diff --check.PYTHONPYCACHEPREFIX=/private/tmp/msb-stack8-pycache python3 -m py_compile scripts/smoke/cli/cow-memory-lifecycle.py.cargo test -p microsandbox-runtime --lib --offline(179 tests; rerun outside the filesystem sandbox for temporary IPC sockets).cd sdk/node-ts && npm run test:unit && npm run typecheck(137 tests).Current transport and integration qualification
Pushed head
5f46d63econtains runtime code tested ateada7e20; the only subsequent change expands the Windows CI filter to run both lifecycle-handoff and stdio-guard regressions. It includes the updated #7 base, libkrun 0.1.35 and the transport fixes. Earlier live results below are revision-scoped.f014: macOS and three Linux transport cohorts passed their normal and combined live matrices. These are not final-head/libkrun 0.1.35 qualification.SnapshotArchiveinvalid-receiver/constructor probes passed against an existing native binary. Initial integrated CodeQL gates were neutral/successful; the baseline alert was not suppressed. Fresh analysis and native CI remain separate checks.Integrated Mac SHA256: runtime
f89c4ec3bf154b7daaf077c8185d98ffe37db02d6843635cc4c54e155a6cc339; agentd3219165916e536620519b8a0a543c53a9449ba07635016153a8193444f2a41cf; firmwareea0d458cdc12a0fa6dac8d192542ddc39717f816da41176582905e31a8bf868c.Performance is mixed: frozen Linux all-load PTY median improved 19.72%, but loaded-ping median increased from 0.066 to 1.649 ms during one upload trial, and small TCP latency also increased. These remain performance follow-ups, not a universal speedup claim.
Broader outstanding qualification remains Windows x86-64, NUMA combinations, physical-memory sharing/reclamation under pressure, broader concurrency/failure injection, and repeated release-build performance measurements.
Confidence Score: 5/5
The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.
No blocking failure remains.
Reviews (6): Last reviewed commit: "build(deps): use published snapshot runt..." | Re-trigger Greptile